MROrchestrator: A Fine-Grained Resource Orchestration Framework for Hadoop MapReduce

Efficient resource management in data centers and clouds running large distributed data processing frameworks like Hadoop is crucial for enhancing the performance of hosted MapReduce applications, and boosting the resource utilization. However, existing resource scheduling schemes in Hadoop allocate resources at the granularity of fixed-size, static portions of the nodes, called slots. A slot represents a multi-dimensional resource slice, consisting of CPU, memory and disk on a machine. In this work, we show that MapReduce jobs have widely varying demands for multiple resources, making the static and fixedsize slot-level resource allocation a poor choice both from the performance and resource utilization standpoints. Furthermore, lack of coordination in the management of multiple resources across the nodes, prevents dynamic slot reconfiguration and leads to resource contention. Towards this end, we perform a detailed experimental analysis of the performance implications of slotbased resource scheduling in Hadoop. Based on the insights, we propose the design and implementation of MROrchestrator, a MapReduce resource Orchestrator framework, that can dynamically identify the resource bottlenecks, and resolve them through fine-grained, coordinated, and on-demand resource allocations. We have implemented MROrchestrator on two 24-node native and virtualized Hadoop clusters. Experimental results with suite of representative MapReduce benchmarks demonstrate up to 38% improvement in reducing job completion times, and up to 25% increase in resource utilization. We further show how popular resource managers like NGM and Mesos when augmented with MROrchestrator can boost their performance.